Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

Users’ visual attention is highly fragmented during mobile interactions but the erratic nature of these attention shifts currently limits attentive user interfaces to adapt after the fact, i.e. after shifts have already happened, thereby severely limiting the adaptation capabilities and user experience. To address these limitations, we study attention forecasting – the challenging task of predicting whether users’ overt visual attention (gaze) will shift between a mobile device and environment in the near future or how long users’ attention will stay in a given location. To facilitate the development and evaluation of methods for attention forecasting, we present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). As a rst step towards a fullyedged attention forecasting interface, we further propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users’ visual scene. We demonstrate the feasibility of forecasting bidirectional attention shifts between the device and the environment as well as for predicting the rst and total attention span on the device and environment using our method. We further study the impact of di erent sensors and feature sets on performance and discuss the signi cant potential but also remaining challenges of forecasting user attention during mobile interactions.